Octopus: A Framework for Cost-Quality-Time Optimization in Crowdsourcing

Managing micro-tasks on crowdsourcing marketplaces involves balancing conƒicting objectives – the quality of work, total cost incurred and time to completion. We present Octopus, the €rst AI agent that jointly manages all three objectives, as a €rst step towards this goal. Previous agents have focused on cost-quality, or cost-time tradeo‚s, limiting their real-world applicability. Octopus is based on a computationally tractable, multi-agent formulation consisting of three components: one that sets the price per ballot to increase/decrease the rate of completion of tasks, another that optimizes each task for quality and a third that performs task selection. We demonstrate that Octopus outperforms existing state-of-the-art approaches in simulation and live experiments, demonstrating its superior performance. We also deploy Octopus on Amazon Mechanical Turk to establish its ability to manage tasks in a real-world, dynamic seŠing. ACM Reference format: Karan Goel, Shreya Rajpal, and Mausam. 2017. Octopus: A Framework for Cost-‹ality-Time Optimization in Crowdsourcing. In Proceedings of ACM Conference, Washington, DC, USA, July 2017 (Conference’17), 10 pages.